Issues in large - scale assessments Issues in Large - scale Assessments

In large-scale assessments such as state-wide testing programs, national sample-based surveys and international comparative studies, there are many steps involved in the measurement and reporting of student achievement. Typically, the steps involved include the following:  Items are developed and test instruments are assembled;  Student samples are selected (for sample-based surveys);  Tests are administered;  Students’ responses are marked (if there are open-ended items);  Test results are scaled and equated (if trend analysis is of interest);  Indicators are estimated (at individual and/or group levels);  Reports are produced and conclusions are drawn. The steps outlined above require expertise from different fields, including subject specialists, sampling statisticians, psychometricians and administrators. The steps are complex, and there is on-going research in all areas. Nevertheless, there are always sources of inaccuracies in each of the steps. It is of interest to identify the factors in the steps that may threaten the validity of the final results. Furthermore, an assessment of the relative magnitude of the threats will place the threats in order of importance. Survey designers can then improve the survey quality by focusing on areas that pose the highest threats to the validity of results. This paper examines a number of factors that can lead to invalid results. In particular, sources of systematic errors pose the greatest threats to surveys. For example, a sample that is not sufficiently representative of the population will give biased results no matter how accurate the measurement process is. On the other hand, measurement disturbances such as differential item functioning may lead to serious bias in the results no matter how representative the sample is (e.g., even if the population is surveyed). This paper discusses the magnitude of various sources of errors with reference to the objectives of assessment programs. A number of examples from large-scale surveys are used to illustrate threats to the surveys and the impact of the threats. The paper concludes by making a Issues in large-scale assessments 6 number of recommendations that could lead to an improvement of the validity of large-scale survey results. Issues in large-scale assessments 7